1. Field
The present disclosure relates generally to a data processing system and more particularly to a system for loading data files. Still more particularly, the present disclosure relates to a system for loading and editing large columnar text files.
2. Background
Aircraft production data is often exported as columnar text data files for analysis such as comparison between logical data and physical data. These files are extremely large and often have a size that is greater than the physical random access memory (RAM) size of most computers. As a result of the large file size, there is limited capability to view and analyze the data. Data analysts or engineers cannot load the data into computer memory to check and verify data integrity and completeness due to the large file size exceeding available memory. Instead, the data must be checked manually.
The manual processes could possibly involve multiple approaches. One approach is to divide the large file into numerous smaller files. The smaller files are then loaded one at a time for editing, verifying, and/or comparing. Finally, all the divided smaller files must be combined back into a large file as a whole data set for further analysis. This approach is extremely time consuming, as the number of divided smaller files typically reaches into the hundreds. Another approach is to create a new specific database for storing the data content of the large file. This approach enables data analysis, such as editing, verifying, and/or comparing data, to be performed, but in a complicated way lacking user-friendly interfaces. This approach requires database administrators to perform the analysis that data analysts and/or data process engineers cannot perform. In addition, the creation and maintenance of an extra database can be costly. Still another approach is to use existing text editors to verify data. Existing text editors, which are able to edit large files, result in poor performance and a loss of the columnar view. These existing text editors instead display the data in continuous rows of characters that must be manually parsed by a human analyst or engineer. This approach cannot perform column-related operations, such as column sorting, column re-ordering, column deletion, and column comparison, for example.
Therefore, it would be advantageous to have a method and apparatus that takes into account one or more of the issues discussed above, as well as possibly other issues.